Fpga-based Rectification and Lens Undistortion for a Real-time Embedded Stereo Vision Sensor

Modern applications such as robot assembling automation, upcoming mobile robot platforms for homes, and car safety features require both, 3D perception and object classification, for navigation and object manipulation [9]. Real-time performance of approximately 30 frames per second is mandatory and a very critical design issue. Embedded stereo vision sensors, consisting of a sensor head and a calculation unit, are very well suited for stereoscopic perception but require huge computational effort. Due to mounting tolerances within the sensor head, resulting in a maximum relative shift and revolution of these two camera images, rectification is absolutely necessary to reduce the matching effort. In our employed method it is done by applying a remap with offline calculated coefficients. The camera lenses also have an impact on the source image, resulting in a distortion in border areas which is reversed by a second remap. By undistorting and rectifying the original camera images, the computational effort for subsequent tasks like stereo matching is dramatically reduced, thus setting a very important step to achieve real-time performance. Approaches implementing lens undistortion and rectification on DSP based platforms or general purpose CPU based platforms are very often insufficient [6]. Considering reduced power usage and small form factors, general purpose CPUs (even with multimedia extensions) are outperformed by Digital Signal Processors (DSPs) and their massive parallel architecture [4]. Also the DSP’s resource usage, including computation and memory transfers for this correction process, is far too high and nearly leaves no evaluation time for high level applications. High memory bandwidths, low memory latencies, and the overall resource consumption inhibit the usage of DSPs for this application [2]. The presented solution to overcome this bottleneck is a dedicated hardware implementation, bypassing the disadvantages of common platforms. Current Field Programmable Gate Array (FPGA) systems deliver enough resources and performance, by reduced form factors to fulfill all necessary requirements [7],[10]. Their tightly coupled memory architecture, and their possibility to flatten designs qualifies them for image processing [11]. Therefore, a generic IP-Core with a simple input/output image interface, a parameter cache interface for the combined undistortion and rectification coefficients and minimized FPGA utilization is designed and tested on an existing platform. By keeping this core generic, it offers a nearly seamless integration and reusability for future projects, too. This paper is organized as follows: Section 3 shortly briefs on the basics and necessities of rectification and undistortion. Sections 4 and 5 present the proposed architecture, describe the target oriented implementation and synthesis results. The conclusion is supplemented with techniques for eliminating the cache architecture by on-the-fly calculation of the remap coefficients.